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ABSTRACT 

This  document  addresses  the  problem  of  defining  the  Measures  Of  Performance 
(MOP)  applicable  to  the  MultiSensor  Data  Fusion  (MSDF)  problem  to  be  tested  in  the 
Advanced  Shipbome  Command  and  Control  Technology  (ASCACT)  testbed.  A  finite  set 
of  MOPs  is  considered  in  order  to  highlight  the  following  three  major  aspects  of  the 
performance:  the  reaction  time,  the  track  quality  and  the  identification  accuracy.  The 
purpose  of  the  performance  evaluation  process  is  to  evaluate  the  ability  of  the  MSDF 
system  to  generate  the  estimated  tactical  picture  that  accurately  reproduce  the  ground 
truth  tactical  picture.  The  evaluation  objective  is  to  quantify  the  potential  divergence  that 
may  result  from  the  many  limitation  factors  affecting  the  performance  of  MSDF  systems. 


r£sum£ 

Ce  document  porte  sur  la  definition  des  mesures  de  performance  applicables  au 
probleme  de  fusion  de  donnees  multicapteur  qui  fera  l'objet  de  tests  sur  le  banc  d'essai 
que  Ton  nomme  ASCACT  (Advanced  Shipboard  Command  and  Control  Technology). 
Des  mesures  de  la  performance  sont  definies  pour  mettre  en  valeur  les  trois  aspects 
suivants:  le  temps  de  reaction,  la  qualite  de  la  piste  et  1'exactitude  de  l'identification.  Le 
but  ainsi  vise  est  de  mesurer  la  capacite  du  systeme  de  fusion  k  dvaluer  l'image  tactique 
et  de  quantifier  la  difference  par  rapport  a  la  realite. 
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EXECUTIVE  SUMMARY 

In  carrying  out  their  missions,  the  Canadian  Patrol  Frigate  (CPF)  crew  is 
bombarded  with  sensor  and  link  information  which  must  be  correlated,  fused  and 
interpreted  in  order  to  arrive  at  some  understanding  of  the  tactical  situation.  At  present, 
the  fusion  of  this  data  is  being  manually  performed  by  the  operators.  In  order  to  cope  with 
the  ever-increasing  flow  and  complexity  of  information,  automation  has  surfaced  as  a 
possible  option  for  the  fusion  of  positional  and  identity  information. 

The  Advanced  Shipboard  Command  And  Control  Technology  (ASCACT)  testbed 
will  be  used  to  investigate  the  adaptation  of  advanced  commercial  data  processing  and 
management  technology  to  enhance  the  ability  of  the  Canadian  Patrol  Frigate  (CPF) 
tactical  crew  to  perform  their  missions.  The  current  phase  of  the  ASCACT  project  puts 
the  emphasis  on  Multi-Sensor  Data  Fusion  (MSDF).  Through  MSDF,  the  system 
developed  under  ASCACT  will  provide  automatic  target  tracking  and  identification  on 
board  the  CPF. 

A  very  important  aspect  of  the  current  phase  of  the  ASCACT  testbed  is  to  assess 
the  performance  of  an  MSDF  demonstration  program.  Unfortunately,  no  widely  accepted 
scheme  for  characterizing  the  performance  of  MSDF  systems  is  currently  in  use.  While 
much  research  is  being  carried  out  to  develop  and  apply  new  MSDF  algorithms  and 
techniques,  little  work  has  been  performed  to  determine  how  well  such  methods  work  or 
to  compare  alternative  methods  against  a  common  problem. 

This  document  addresses  this  problem  by  defining  the  Measures  Of  Performance 
(MOP)  applicable  to  the  MSDF  problem.  A  finite  set  of  MOPs  is  considered  in  order  to 
highlight  the  following  three  major  aspects  of  the  performance:  the  reaction  time,  the 
track  quality  and  the  identification  accuracy.  The  purpose  of  the  performance  evaluation 
process  is  to  measure  the  ability  of  the  MSDF  system  to  generate  the  estimated  tactical 
picture  that  accurately  reproduce  the  ground  truth  tactical  picture.  The  evaluation 
objective  is  to  quantify  the  potential  divergence  that  may  result  from  the  various 
limitation  factors  affecting  the  performance  of  MSDF  systems. 

Many  CPF  enhancement  projects  regarding  automation  for  its  Command  and 
Control  Information  System  (CCIS)  have  been  put  forward  recently  in  the  framework  of 
the  Frigate  Life  Extension  program  (FELEX).  Within  this  framework,  ASCACT,  a  major 
DND  project  (D6195)  is  crucial.  The  research  and  results  described  in  this  memorandum 
are  being  directly  used  for  ASCACT  in  order  to  evaluate  if  an  automated  MSDF  system 
can  enhance  the  ability  of  the  CPF  tactical  crew  to  carry  out  their  missions. 
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1.0  INTRODUCTION 

The  objective  of  the  Advanced  Shipbome  Command  And  Control  Technology 
(ASCACT)  project  is  to  improve  the  shipboard  data  processing  capability  of  naval 
Command  and  Control  Information  Systems  (CCISs)  by  developing  an  Advanced 
Development  Model  (ADM)  multiple  processor  testbed.  This  testbed  will  be  used  to 
investigate  the  adaptation  of  advanced  commercial  data  processing  and  management 
technology  for  military  applications.  The  testbed  will  also  be  used  to  explore  various 
CCIS  concepts,  such  as  Real-Time  Distributed  Database  Management  Systems 
(RTDDBMS),  Multi-Sensor  Data  Fusion  (MSDF),  Situation  and  Threat  Assessment 
(STA),  Resource  Management  (RM),  etc.  Indeed,  the  ASCACT  testbed  will  make  it 
possible  to  investigate  on  any  combination  of  the  requirements  mentioned  above.  Hence, 
the  testbed  will  be  built  as  a  versatile  tool  that  will  be  used  to  develop  the  CCIS 
applications  of  interest. 

The  ASCACT  stand-alone  integration  testbed  described  in  Ref.  1  is  a  subset  of  the 
overall  testbed  to  be  developed  under  the  ASCACT  project.  The  expression  "stand-alone" 
refers  to  the  fact  that  this  testbed  is  autonomous  (i.e.,  it  is  not  tied  to  another  system  while 
being  used).  The  current  phase  of  the  ASCACT  project  puts  the  emphasis  on  a  related 
shipboard  CCIS  concept,  i.e.,  Multi-Sensor  Data  Fusion  (MSDF).  MSDF  refers  to  the 
process  of  amalgamating  multiple  dependent  sensor  data  sets,  while  periodically 
providing  relevant  target  estimates  based  on  all  this  information.  The  MSDF  objective  in 
the  specific  context  of  the  ASCACT  project  is  to  enhance  the  ability  of  the  Canadian 
Patrol  Frigate  (CPF)  tactical  crew  to  perform  their  missions.  In  carrying  out  their 
missions,  the  CPF  crew  is  bombarded  with  sensor  information  which  must  be  correlated, 
fused  and  interpreted  in  order  to  arrive  at  some  understanding  of  the  tactical  situation.  At 
present,  the  fusion  of  this  data  is  being  manually  performed  by  the  operators.  Through 
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MSDF,  the  system  developed  under  this  project  will  provide  automatic  target  tracking 
and  identification  on  board  the  CPF. 

This  document  defines  the  Measures  Of  Performance  (MOP)  for  the  MSDF 
demonstration  program  of  the  ASCACT  stand-alone  integration  testbed.  A  finite  set  of 
MOPs  is  considered  in  order  to  highlight  the  following  three  major  aspects  of 
performance: 

-  reaction  time; 

-  track  quality; 

-  identification  accuracy. 

The  main  contents  of  this  document  is  distributed  over  two  sections.  Chapter  2 
addresses  the  problem  of  how  to  measure  the  performance  of  an  MSDF  system,  while  in 
Chapter  3  the  definitions  of  the  MOPs  for  the  MSDF  Demonstration  program  are  given 
(Ref.  1). 

This  work  was  carried  out  at  DREV  between  January  1993  and  December  1995 
under  Project  lae  "Shipboard  Command  and  Control,  Work  Unit  WUlae25  "Support  to 
the  ASCACT  Working  Group". 


2.0  PROBLEM  DESCRIPTION 

One  aspect  of  the  Performance  Evaluation  mode  of  the  ASCACT  Testbed  (Ref. 
1)  is  to  assess  the  performance  of  the  MSDF.  Unfortunately,  no  widely  accepted  scheme 
for  characterizing  the  performance  of  MSDF  systems  is  currently  in  use.  While  much 
research  is  being  performed  to  develop  and  apply  new  MSDF  algorithms  and  techniques, 
little  work  has  been  carried  out  to  determine  how  well  such  methods  work  or  to  compare 
alternative  methods  against  a  common  problem. 

Figure  1  illustrates  the  main  elements  of  the  MSDF  performance  evaluation 
concept.  The  ground  truth  picture  represents  the  real  composition  and  status  of  a  scenario 
of  tactical  interest.  The  objective  of  the  overall  system,  whose  performance  is  affected  by 
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factors  such  as  energy  propagation  effects,  is  to  keep  an  awareness  of  the  external 
situation  as  efficiently  and  accurately  as  possible.  During  its  operation,  the  overall  system 
generates  both  a  measured  tactical  picture  (i.e.,  the  output,  not  explicitly  shown  in  Fig.l, 
of  the  sensing  process)  and  an  estimated  picture  (whose  quality  should  in  principle  be 
superior  to  that  of  the  measured  picture).  The  purpose  of  the  performance  evaluation 
process  is  to  measure  the  ability  of  the  MSDF  system  to  generate  the  estimated  tactical 
picture  that  accurately  reproduce  the  ground  truth  tactical  picture.  The  evaluation 
objective  is  to  quantify  the  potential  divergence  that  may  result  from  the  various 
limitation  factors  affecting  the  performance  of  MSDF  systems. 

2.1  Ground  Truth  Tactical  Picture 

The  ground  truth  tactical  picture  is  used  as  a  basis  for  comparison  with  the 
measured  and  estimated  tactical  pictures.  It  depicts  the  known  activities  (i.e.,  position, 
kinematic  behavior,  emissions,  and  identity)  of  a  known  number  Ng  of  real,  distinct 
targets  in  a  given  area  of  interest.  A  target  may  be  a  plane,  ship,  missile,  etc.  Targets 
possess  defined  kinematic  and  non-kinematic  properties.  The  trajectory  of  a  target 
typically  summarizes  its  kinematic  properties.  The  target  type  or  category,  its  allegiance, 
nationality,  threat  level  and  specific  identification  are  examples  of  non-kinematic 
properties.  The  ground  truth  targets  progress  in  time  and  space  according  to  a  well- 
defined  scenario  that  also  includes  the  characteristics  of  the  target  emissions. 

The  MSDF  system  has  to  operate  in  the  four-dimensional  world  of  space  and 
time.  In  the  ASCACT  testbed,  this  world  is  stored  in  the  Performance  Analysis  DataBase 
(PADB)  (Ref.  1).  To  assess  the  performance,  the  ASCACT  user  must  specify  a  Volume 
Of  Interest  (VOI)  of  the  PADB.  In  the  development  of  performance  evaluation  plans,  one 
must  determine  which  targets  have  penetrated  the  VOI.  The  test  scenarios  for  the  MSDF 
system  must  be  larger  than  the  VOI,  so  that  targets  will,  throughout  the  scenario,  both 
enter  and  exit  the  VOI. 
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Figure  2  is  an  illustration  of  the  concept  of  VOI  discussed  above.  Since  one  is 
almost  certainly  concerned  with  a  dynamic  environment,  one  needs  to  take  time  into 
account  as  well.  This  is  also  shown  in  Fig.  2.  The  Time  Interval  Of  Interest  (TIOI)  for 
performance  evaluation  is  any  suitable  interval  during  which  one  is  interested  in 
comparing  the  measured  and  estimated  tactical  pictures  with  the  ground  truth  picture. 

Hence,  for  the  purpose  of  MSDF  system  performance  evaluation  in  ASCACT, 
reality  or  ground  truth  is  represented  by  only  the  Ng  objects  of  interest  progressing  within 
the  VOI  during  the  TIOI.  It  is  assumed  that  the  MSDF  evaluator  knows  at  any  time  all  the 
characteristics  and  behavior  of  the  set  of  targets  which  are  actually  part  of  ground  truth. 
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FIGURE  -1  -  The  MSDF  performance  evaluation  concept 
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FIGURE  -  2  -  Definition  of  the  VOI  and  the  TIOI 


2.2  Hierarchy  of  Measures 

To  assist  the  Command,  Control  and  Information  (CCI)  community  in  choosing 
the  most  appropriate  measures,  the  Military  Operations  Research  Society  (MORS) 
developed  a  four-level  hierarchy  of  measures  (Refs  2-3).  The  lowest  level  measures  are 
called  dimensional  parameters,  followed  by  the  Measures  Of  Performance  (MOPs)  and 
Measures  Of  Effectiveness  (MOEs).  The  highest  level  measure  is  called  the  Measure  Of 
Force  Effectiveness  (MOFE). 

The  dimensional  parameters  are  the  properties  or  characteristics  of  a  physical 
entity  whose  values  determine  system  behavior,  and  the  structure  under  consideration, 
even  when  the  entity  is  at  rest.  This  is  not  enough  for  the  ASCACT  testbed.  The  measures 
of  Performance  are  related  to  the  dimensional  parameters,  but  they  measure  aspects  of  the 
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system  behavior.  This  is  the  appropriate  level  for  the  MSDF  performance  evaluations  to 
be  performed  in  the  ASCACT  testbed.  Measures  of  Effectiveness  are  used  to  measure 
how  the  CCIS  performs  its  functions  within  an  operational  environment.  The  measures  of 
Force  Effectiveness  allow  one  to  determine  how  the  CCIS,  and  the  force  of  which  it  is  a 
part,  performs  their  mission.  Clearly,  the  last  two  measures  are  too  high  level  for  the 
MSDF  Demonstration  Program. 

3.0  Measures  of  Performance  for  the  MSDF  Demonstration  Program 

The  MOP  problem  of  an  MSDF  system  can  be  stated  as  follows.  Given  the  output 
of  an  MSDF  algorithm,  how  does  one  determine  the  "performance"  of  the  algorithm?  The 
answer  depends  on  one's  purpose  and  leads  to  distinctions  between  types  of  MOPs.  The 
type  of  MOPs  considered  here  are  those  which  measure  how  well  an  MSDF  algorithm  is 
performing  against  tactically  crucial  parameters  (localization  radial  miss  distance,  for 
example).  These  are  typically  of  great  interest  to  the  user  community. 

Praticality  dictates  that  several  different  MOPs  should  be  defined  to  cover  several 
aspects  of  performance  at  a  time.  The  MSDF  algorithms  are  typically  very  complex, 
highly  integrated  algorithms  whose  different  parts  interact  with  each  other  in  highly 
nonlinear  and  unpredictable  ways.  As  a  result,  it  is  entirely  possible  (and  in  fact  not 
unusual)  that  one  aspect  of  performance,  as  measured  by  one  specific  MOP,  can  be 
improved  only  at  the  cost  of  decreasing  some  other  aspect.  Unfortunately,  it  is  also 
possible  that  the  aspect  of  performance  which  has  been  decreased  will  go  undetected  by 
the  existing  suite  of  MOPs.  For  the  sake  of  this  project,  a  finite  set  of  MOPs  is  considered 
in  order  to  illuminate  the  following  three  major  aspects  of  performance: 

-  reaction  time; 

-  track  quality; 

-  identification  accuracy. 
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3.1  Reaction  Time 

The  reaction  time  is  defined  as  the  time  taken  by  the  MSDF  system  to  output  a 
useful  tracking  or  identification  result  about  objects  entering  in  the  VOI.  The  following 
set  of  parameters  is  needed  to  compute  the  reaction  time  MOPs: 

•  time  at  target  detection; 

•  time  at  track  confirmation; 

•  time  of  positive  identification; 

•  time  of  track  deletion. 

When  a  target  enters  the  VOI,  it  is  detected,  tracked  and  identified.  The  distance  at 
which  a  target  is  tracked  and  identified  is  of  capital  importance  for  the  survivability  of  the 
ship.  Figure  3  represents  (a)  the  detection,  (b)  the  target  ID  different  from  “unknown”,  (c) 
the  “correct”  target  ID  and  (d)  the  “correct  positive”  target  ID  stages  of  a  target 
penetrating  the  surveillance  zone. 
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FIGURE  3  -  Targets  entering  the  VOI 

The  following  MOPs  shall  be  calculated  to  assess  the  MSDF  reaction  time  performance. 

3.1.1  Time  in  VOI  Prior  to  Detection 

This  MOP  measures  the  relative  reaction  time  of  the  individual  sensors  with 
respect  to  their  capacity  to  first  detect  targets.  The  following  elements  need  to  be 
retrieved  from  the  PADB: 

-  a  list  of  sensors  with  their  associated  contacts; 

-  for  each  contact,  the  associated  time  and  the  true  target  origin; 

-  from  ground  truth,  the  simulation  time  when  each  target  is  entering  the  VOI. 
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3.1.2  Time  from  Detection  to  Confirmation 

This  MOP  determines  the  time  interval  between  the  first  detection  by  a  given 
sensor  that  initiates  a  track  and  the  time  when  the  MSDF  Software  confirmed  the  track. 
The  following  elements  need  to  be  retrieved  from  the  PADB: 

-  a  list  of  sensors  with  their  associated  contacts; 

-  for  each  contact,  the  associated  time  and  the  true  target  origin; 

-  a  list  of  all  confirmed  tracks  with  the  associated  time  and  the  true  target  origin. 

3.1.3  Time  of  Positive  Identification 

This  MOP  evaluates  the  time  taken  by  the  MSDF  system  to  correctly  and 
positively  identify  a  target  penetrating  in  the  VOI.  This  MOP  can  be  computed  by 
monitoring  the  belief  of  the  identification  propositions  in  the  PADB,  and  by  extracting 
the  time  when  the  belief  exceeds  a  threshold  specified  by  the  user  (typically  80%).  A 
positive  identification  is  defined  as  a  proposition  whose  belief  is  above  this  threshold. 
The  following  elements  need  to  be  retrieved  from  the  PADB: 

-  the  list  of  all  confirmed  tracks; 

-  the  list  of  the  identification  propositions,  with  their  associated  beliefs; 

-  the  true  identity  of  the  target  and  the  time  this  target  has  penetrated  the  VOI. 

The  identification  time  can  also  be  shown  with  respect  to  the  track  confirmation  time. 

3.1.4  Time  for  Track  Deletion 

This  MOP  determines  the  time  taken  by  the  MSDF  system  to  delete  a  track  when 
the  target  leaves  the  VOI.  The  following  elements  need  to  be  retrieved  from  the  PADB: 

-  a  list  of  all  confirmed  tracks  and  the  true  target  origin; 

-  from  ground  truth,  the  simulation  time  when  the  target  leaves  the  VOI. 
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3.2  Track  Quality 

The  measures  of  individual  track  quality  focus  on  three  aspects  of  the  tracking 
process:  (1)  track  estimation,  (2)  track  purity  (correlation),  and  (3)  track  management. 
There  is  a  close  relationship  between  track  estimation,  correlation,  and  management 
performance  since  the  systems  that  correlate  poorly  will,  in  general,  also  track  poorly,  and 
the  systems  that  track  targets  poorly  will  generally  result  in  an  increased  number  of  failed 
correlations  and  miscorrelations  leading  to  poor  track  management  performance. 

3.2.1  Track  Estimation 

The  geopositional  MOPs  determine  how  well  ground  truth  targets  are  being 
physically  localized  with  respect  to  position  and  velocity  (and  acceleration,  if  it  is  a  state 
variable).  The  time  history  of  the  state  covariance  matrices  also  makes  it  possible  to  study 
the  behavior  of  the  MSDF  errors  for  different  target  maneuvers. 

The  target  state  estimation  function  transforms  sensor  measurements  into 
estimates  of  the  target's  state  (usually  the  target's  trajectory  described  by  position, 
velocity,  acceleration,  etc.)  and  their  corresponding  state  estimation  error  covariances. 
The  difference  (or  distance)  between  the  state  estimate  and  the  true  state  defines  the  state 
estimation  error.  One  straightforward  measure  of  state  estimation  accuracy  is  the  error 
magnitude.  Direct  measures  include  position  error,  velocity  error,  etc.,  defined  in  the 
usual  manner  as  the  magnitude  of  the  position  component,  velocity  component,  etc.,  of 
the  state  estimation  error  vector. 


The  following  three  MOPs  are  used  to  estimate  the  track  estimation  errors: 


P501928.PDF  [Page:  21  of  42] 


UNCLASSIFIED 

11 


3.2.2  Radial  Miss  Distance 

At  a  given  time,  if  x,  is  the  estimated  position  of  the  track  t,  and  xg  the  actual 
position  of  ground  truth  target  g,  then  the  radial  miss  distance  is  just  the  Pythagorean 
distance  between  them: 


RMD(/,g)  = 


xt~x* 


(3.1) 


where  "I  I"  denotes  the  usual  Pythagorean  distance  in  3D-space.  The  following  elements 
need  to  be  retrieved  from  the  PADB: 

-  the  position  components  of  the  estimated  state  vector; 

-  the  position  components  of  the  ground  truth  state  vector; 

-  the  corresponding  time. 

3.2.3  State  Estimation  Error 


Given  the  assignment  of  tracks  to  truth  for  various  evaluation  times,  the  accuracy 
of  the  track  estimate  can  be  evaluated.  The  type  of  estimates  (predicted,  filtered,  or 
smoothed)  must  be  selected  as  well  as  the  evaluation  times.  For  each  of  the  track-target 
pairs  that  are  associated,  one  computes  the  state  estimation  error  as  the  difference 
between  the  true  target  state  x(k )  and  the  track  estimated  state  at  time  k  given 
measurements  up  to  time  n: 


x(k\n)  =  x(k)  —  x(k\n)  (3.2) 

The  estimation  error  is  computed  for  each  component  of  the  state  vector  including 
velocity  and  acceleration  as  opposed  to  RMD  which  is  only  a  positional  MOP.  The 
following  elements  need  to  be  retrieved  from  the  PADB: 

-  the  estimated  state  vector; 

-  the  ground  truth  state  vector; 
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-  the  corresponding  time. 

3.2.4  Accuracy  of  the  Filter  Calculated  Covariance 

The  general  idea  behind  this  MOP  is  to  compare  the  covariance  matrices  between 
different  variant  of  the  MSDF  system  (Ref.  4).  Each  sensor  measurement  or  estimated 
parameter  has  associated  uncertainties.  These  uncertainties  can  be  represented  for  a  track 
as  ellipses  (i.e.,  the  covariances  are  converted  into  error  ellipses  for  a  given  confidence 
level)  in  the  X-Y  space  for  positional  data  and  in  speed-bearing  space  for  velocity  data. 
The  accuracy  of  the  tracks  can  be  evaluated  considering  the  area  of  those  ellipses  and 
comparison  can  be  made  between  MSDF  systems.  This  comparison  will  provide  an 
approximative  evaluation  of  the  relative  performance  of  the  tracking  algorithms. 

The  ellipse  parameters  are  computed  via  the  eigenvalues  of  the  covariance  matrix 
(considering  only  the  position  covariance  matrix  or  the  velocity  covariance  matrix),  and 
the  confidence  threshold  ( Pe )  of  the  ellipse.  Pe  is  the  probability  determined  by  the  user 
that  the  state  vector  lies  inside  the  ellipse.  The  position-velocity  cross  terms  of  the 
covariance  matrix  are  not  considered.  The  semi  axes  of  the  ellipse  are  given  by  the 
following  equation: 


where 


a  =  ^Kkx ,  b  =  *JkX2 


\,2  ~  0-5  cu  d"  cn  —  "\/(ci  I  c22)  +  4(c12c21) 


with  the  C(j,  representing  the  covariance  matrix  elements,  and 


(3.3) 


(3.4) 


K  =  -21n(l  -  Pe) 


(3.5) 
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A  direct  comparison  of  the  error  ellipses  is  performed  by  computing  the 
percentage  difference  between  the  error  ellipse  areas  of  the  two  systems.  The  following 
elements  need  to  be  retrieved  from  the  PADB: 

-  the  estimated  state  vector  and  its  covariance  matrix; 

-  the  corresponding  time. 

3.2.5  Track  Purity  (Misassociation) 

Track  purity,  a  concept  coined  by  Mori,  et.  al  (Ref.  5)  assesses  the  percentage  of 
correctly  associated  measurements  in  a  given  track,  and  so  evaluates  the 
association/tracking  performance.  This  MOP  is  not  explicitly  dependent  on  detection 
performance,  but  it  is  dependent  on  the  settling  of  association  gates  (and  thus  a  function 
of  the  average  innovation  standard  deviation  which  depends  on  Pa)-  It  is  also  dependent 
on  the  target  density.  Track  purity  determines  the  consistency  with  which  a  track  is 
updated  with  measurements  from  a  single  target  or  a  set  of  targets. 

Correlational  local  MOPs  such  as  track  purity  determine  how  well  the  tracks  in  an 
MSDF  system  are  being  associated  with  measurements  of  ground  truth  targets.  The  track 
purity  MOP  is  based  on  the  calculation  of  a  so-called  confusion  matrix  C  for  which  the 
elements  C;-,-  are  constructed  by  counting  reports.  Given  a  single  hypothesis  h  with  tracks 
tj, ...,  tb  and  a  set  of  ground  truth  targets  gi, ga  it  has  the  form 
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*  1 

Targets 

X  2 

X  a 

t  , 

C0! 
c , 

C02 
c , 

C0a 

c 

1 

11 

12  — 

la 

1  2 

C21 

C  22 

C2a 

Tracks 

• 

t  , 

c. 

c 

c 

b 

bl 

b2 

ba 

Here,  Q  is  the  number  of  reports  originating  from  ground  truth  target  g,  which 
were  assigned  to  track  tj  (i  =  1, a;j  =  1, b)  by  the  MSDF  algorithm.  Also,  C0i  (the 
"ambiguity  vector")  consists  of  the  number  of  reports  which  could  not  be  assigned  to  any 
track  (i  =  1, ...,  a).  When  Cji  is  large,  a  strong  association  between  tj  and  g,  is  implied. 

Track  purity  is  defined  as  the  percentage  of  correctly  associated  measurements 
contained  in  a  given  track.  The  purity  of  the  track  tj  in  the  hypothesis  h  is  defined  as  the 
normalized  value  of  the  largest  element  in  the  row  defined  by  tji 


max  C„ 


l</<3 


Zc, 

1=1 


More  generally,  the  total  track  purity  of  the  hypothesis  is: 

j 


(3.6) 


(3.7) 


where  the  row  weights  qj  are  defined  by: 
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a 

Ic, 

9,=~T -  (3-8) 

I  Zc, 

«=1  7=1 

The  following  elements  need  to  be  retrieved  from  the  PADB: 

-  the  list  of  tracks  and  the  list  of  associated  contacts; 

-  from  ground  truth,  the  list  of  targets  and  their  contacts; 

-  the  corresponding  time. 

3.2.6  Track  Management  Statistics 

The  track  maintenance  statistics  are  the  number  of  potential  tracks,  the  number  of 
tentative  tracks,  the  number  of  confirmed  tracks  and  the  number  of  confirmed  tracks  that 
will  not  later  be  deleted  early.  These  numbers  can  be  counted  from  the  PADB. 

3.3  Identification  Accuracy 

The  purpose  of  a  classification  MOP  is  to  score  how  well  an  MSDF  algorithm  is 
identifying  targets.  It  is  assumed  that  the  MSDF  algorithm  whose  performance  is  to  be 
evaluated  has  an  identification-classification  capability.  It  is  also  assumed  that  this 
classification  system  is  based  on  a  Dempster-Shafer  type  of  representation  scheme.  That 
is,  if  t  is  a  track,  then  the  MSDF  algorithm's  estimate  of  the  identity  of  t  is  assumed  to  be 
a  "body  of  evidence"  Bt  of  the  following  form: 

Bt  ={(P1,m1),(P2,/n2),...,(pi,mJ)}  (3.9) 

Here,  each  P,  is  a  logical  well-formed  formula  which  makes  a  statement  concerning  the 
identity  of  t  (list  of  propositions)  ;  and  m*  (belief  associated  with  each  proposition)  is 

such  that  mi  for  all  i  =  1,  s  and  Xi=im<  =  Each  pair  (Pb  ntj)  thus  represents  a  model 
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Pi  of  the  identity  of  t,  together  with  an  estimate  m,  of  how  likely  P,  represents  t  (in 
comparison  with  the  other  models  in  Bt). 

In  particular,  given  such  a  representation  scheme,  ground  truth  targets  are 
represented  by  propositions  of  the  form  Bg  =  {(Eg,  1)}  where  Eg  =  "x  is  g"  and  where  we 
set  x  =  t  if  g  is  being  compared  with  track  t.  Likewise,  the  so-called  null  proposition  has 
the  form  0  =  "xe  U"  where  U  represents  the  universe  of  all  targets  which  could  have 
been  part  of  ground  truth 

In  what  follows,  ”g”  can  refer  either  to  the  identity  of  a  specific  target  (if  the 
classification  system  has  this  capability)  or  to  the  NATO  class  of  a  target  (if  the 
classification  system  is  capable  of  no  finer  resolution).  Expressed  in  these  terms,  the  basic 
metrical  question  for  attributes  then  is  as  follows:  Given  a  ground  truth  target  g  and  a 
track  t,  how  well  or  poorly  does  Bt  describe  the  identity  Eg  of  gl 

3.3.1  Bayesian  Percent  Attribute  Miss  (BP AM) 

The  BP  AM  MOP  dnayes(  t,  g)  is  Bayesian  in  the  sense  that  it  can  be  used  only  if  the 
MSDF  algorithm  evaluator  has  knowledge  of  the  natural  frequencies  of  targets. 
Specifically,  two  things  must  be  known: 

1)  the  universe  U  of  all  targets  which  could  have  been  part  of  ground  truth, 

2)  for  each  logical  proposition  P  supplied  by  the  classification  algorithm,  N(P)  = 
number  of  targets  in  U  which  satisfy  P.  In  particular,  we  set  N  =  N(U). 

For  example,  such  knowledge  typically  requires  that  the  evaluator  has  available  in  PADB 
the  relevant  attributes  of  interest  of  the  targets  of  possible  interest  (i.e,  ground  truth). 

Given  these  assumptions,  the  BP  AM  MOP  is  defined  by: 
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where 


and 


d  Bayes  (t,g)  =  100^(l  +  |z?J2)-a(fl,,£,) 


B,\ 2  = 


/.y=i 


(X(Bt,Eg) 


N(Pt  AEg) 
N(Pt) 


(3.10) 


(3.11) 


(3.12) 


Here,  "A  "  denotes  the  logical  AND  operation.  The  BPAM  MOP  has  the  following  major 

properties: 

1)  0%  <  dBayes(t,g)  <  100% 

2)  d Bayes  (*’  #)  =  0%  if  0nly  if  *  =  8 

3)  dBayes{t,g)  =  100%  if  and  only  if  t  =  g'  for  some  ground  truth  target  g' 
distinct  from  g 

4)  dBayes(t,g)  =  50%  if  there  is  an  equal  likelihood  that  t  is  g  or  g',  where 
g'  is  some  other  target  distinct  from  g 

5)  dBayes(t,g)  =  71%  (that  is,  when  the  evidence  is  totally  inconclusive, 
total  ignorance  =100/^2). 

The  following  elements  need  to  be  retrieved  from  the  PADB: 

-  the  list  of  tracks  and  the  list  of  associated  propositions; 

-  from  ground  truth,  the  list  of  targets  identification  and  their  attributes; 

-  the  corresponding  time. 
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3.3.2  Identification  Range 

When  a  target  enters  the  VOI,  it  is  eventually  detected,  and  then  later  identified. 
The  distance  at  which  a  target  is  identified  can  be  of  capital  importance  for  the 
survivability  of  the  ownship.  The  identification  is  measured  in  terms  of  validity  (i.e., 
correct  ID),  certainty  (i.e.,  ID  confidence)  and  efficiency  (i.e.,  number  of  propositions, 
number  of  platforms  in  database,  etc.)  (Ref.  1). 

3.3.3  Range  of  "Unknown"  Identification 

This  MOP  evaluates  the  range  at  which  the  identification  of  the  target  becomes 
different  from  "unknown".  A  target  is  considered  to  have  an  unknown  identification  when 
no  proposition's  belief  for  that  target  is  above  a  certain  threshold  fixed  by  the  user.  This 
MOP  can  thus  be  computed  by  monitoring  the  belief  of  the  identification  propositions  for 
that  target,  and  by  taking  note  of  the  moment  when  a  belief  exceeds  the  threshold.  The 
range  of  the  target  at  this  particular  moment  can  be  extracted  from  the  history  of 
positions. 

The  identification  range  can  be  evaluated  individually  for  each  track,  or  an 
average  for  all  the  tracks  can  be  formed.  The  average  of  all  the  tracks  will  indicate  the 
range  where  the  targets  are  generally  considered  as  "unidentified".  The  following 
elements  need  to  be  retrieved  from  the  PADB: 

-  the  list  of  propositions  and  their  associated  beliefs; 

-  the  corresponding  time. 

3.3.4  Range  of  "Correct"  Identification 

One  can  re-evaluate  the  previous  MOP  considering  the  true  identities  of  the 
targets.  For  example,  one  can  compute  the  (average)  range  at  which  the  targets  are 
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correctly  identified.  This  is  done,  for  each  individual  target,  by  finding  the  first 
appearance  of  its  real  ID  in  the  elements  of  the  proposition  for  this  target. 

The  following  elements  need  to  be  retrieved  from  the  PADB: 

-  the  list  of  propositions  and  their  associated  beliefs; 

-  from  ground  truth,  the  real  ID  of  the  targets; 

-  the  corresponding  time. 

3.3.5  Range  of  "Positive"  Identification 

This  MOP  evaluates  the  certainty  associated  with  the  identification  of  a  target 
(typically  given  by  a  Dempster-Shafer  algorithm).  A  positive  identification  is  defined  as  a 
proposition  whose  belief  is  above  a  certain  threshold  determined  by  the  user.  Hence  this 
MOP  gives  the  range  where  a  given  confidence  threshold  is  exceeded.  This  threshold 
needs  to  be  determined  in  real  situations,  but  a  representative  value  can  be  fixed  by  the 
user  at  80%.  The  following  elements  need  to  be  retrieved  from  the  PADB: 

-  the  list  of  propositions  and  their  associated  beliefs; 

-  from  ground  truth,  the  real  ID  of  the  targets; 

-  the  corresponding  time. 


4.0  CONCLUSION 

This  document  addressed  the  problem  of  defining  the  Measures  Of  Performance 
(MOP)  applicable  to  the  MultiSensor  Data  Fusion  (MSDF)  problem  to  be  tested  in  the 
Advanced  Shipbome  Command  and  Control  Technology  (ASCACT)  testbed.  A  finite  set 
of  MOPs  were  defined  in  order  to  highlight  three  major  aspects  of  the  performance:  the 
reaction  time,  the  track  quality  and  the  identification  accuracy. 

The  main  elements  of  an  MSDF  performance  evaluation  concept  have  been 
defined.  These  elements  are:  the  ground  truth  picture,  the  volume  of  interest,  the 
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measurement  process,  the  MSDF  system  and  finally  the  estimated  picture.  A  more 
general  discussion  on  the  hierarchy  of  MOPs  was  provided  as  well  in  order  to  choose  the 
appropriate  level  of  the  MOPs  for  the  MSDF  demonstration  program  in  ASCACT. 

A  finite  set  of  approximately  15  MOPs  have  been  defined  to  quantify  the  potential 
divergence  between  the  MSDF  estimated  tactical  picture  and  the  ground  truth  tactical 
picture.  These  MOPs  are  being  used  for  ASCACT  in  order  to  evaluate  if  an  MSDF 
system  can  enhance  the  ability  of  the  CPF  tactical  crew  to  perform  their  missions. 
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